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Abstract. The paper presents a novel method of volumetric reconstruction of transient destructive processes 
using remote sensing by a group of unmanned aerial vehicles. The study is based on the most common class of such 
processes like forest fires, where a fire front is a determinant, and its propagation reflects the dynamics of the process. 
The effects of wind, smoke and fire, turbulence and vibration, interference, distortion, and obstacles lead to uncertainty 
of observations, to overcome which fuzzy sets, soft sets and gray numbers were combined. A spatial model based on a 
recursive eight-fold subdivision of space as well as on a hierarchical structure of virtual cells is proposed, which 
allowed to resolve the contradictions between the accuracy and rate of reconstruction. The set of possible states of 
virtual cells is determined and the algorithm of their classification based on the use of a five-channel image recognition 
system containing infrared, two main, and two additional optical channels is proposed. An algorithm for calculating a 
3D observation vector, presented by an array of confidence vectors, is proposed, which can be used to determine the 
gray fuzzy state of virtual cells allowing a combination of observations from different observers and refining them 
sequentially. The terrain where the process evolves is represented by a soft gray fuzzy set of virtual cells, which belong 
to a specific state at the consideration time, allowing identification of convincing, uncertain, suspicious, and negative 
components. The first one defines a stable core of the fire front while the second one represents its variation caused by 
uncertainty. The proposed method allows the reconstruction of transient spatially distributed processes of other classes, 
smoothing the effects of distortions and noise and ensuring acceptable performance. 


Keywords: unmanned aerial vehicle, multi-view observation, uncertainty, image processing, fire front, virtual 
boxes, grey fuzzy soft set. 
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Anotauis. Y cratti mpescTaBieHo HOBHM MeTOX OO’eMHOi peKOHCTpyKIii WBUAKOMIMHHUX pyMHIBHUXx 
TIpoyeciB 3a MOMOMOTOIO JMCTaHWiMHOrO 30HAYBaHHA TpyMoro Oe3MiIOTHHX JTaIbHUX alapaTis. locus pKeHHA 
BHKOHaHO Ha MpHKati HaliMmowiMpeHiwioro Kacy TaKHX MpoleciB - JICOBHX MOxKexK, Be MpoOHT BOrHIO € 
JeTepMiHaHTOM, a Moro pyx BiqoOpaxae WHHaMiky mpouecy. Brus BiTpy, WHMy Ta BOrHIO, TypOyJIeHTHICTh Ta 
BiOpallii, MepelIKOAU, BUKPHBIICHHA Ta CIOTBOPeHHA UPH3BOAATb IO HeEBH3HAYeCHOCTI CHOCTepexKeHb, WIA MOOMAHHA 
AKOI Oyo 06’€4HAHO HEYITKI HM M’AKi MHOXKMHU Ta «Cipl» 4McIa. 3almpOMOHOBaHO MpOCTOpOBy MOJelIb, 3ACHOBaHY Ha 
PeKypCHBHOMYy BOCbMHeJIeMeHTHOMY MOI MpocTopy Ta iepapxi4Hili CTpyKTypi BIPTyasIbHUX KOMIPOK, WO JO3BOJIMIIO 
BUPIWHTH MpOTHpiy4yd MK TOUHICTIO Ta UIBHAKICTIO peKOHCTpyKuii. Bu3sHa4eHO MHOXKMHY MOXKJIMBHX CTAaHIB 
BIpTYaJIbBHHX KOMIPOK Ta 3aIIpOMOHOBaHO aJIrOpuTM Ix Klacudikatli, 3aCHOBaHHi Ha BUKOPHCTaHHl U'ATHKaHaJIbHOi 
CHCTeMH pO3IIi3HaBaHHA 300paxKeHb, IO MICTHTb iHpavepBOHHi, Ba OCHOBHUX i Ba AOAaTKOBI ONTHUHI Kanan. 
3anponoHOBaHO aJITOpHTM, WO OOUYMCIIOE TpHBHMIpHHit BeKTOp CHOCTepexKeHHA, MOaHvi MaCHBOM BeKTOPiB 
BIICBHEHOCTI, 3a JOMOMOFO!0 AKOTO BU3HAYAIOTH CipHit HEYITKUM CTaH BIPTyaJIbHUX KOMIPOK, WO AO3BOJIAE NOE AHYBATH 
cnocTepexKeHHA BI pi3HHX cHOcTepirayiB Ta WOCciOBHO ix yTouHToBaTH. MicieBicTb, We Po3BHBaeTbCA MpouLec, 
TOaHO M’AKOIO CIPOIO HeYITKOIO MHOXKHHOIO BIPTYAIbHHX KOMIPOK, AKi BIHOCATBCA TO MeBHOTO CTaHy Ha MOMCHT 
posriady, WO TO3BOIAe BH3HAYHTH MepeKOHMBUM, HEBH3HAYeHHH, WiO3piWui Ta HeraTHBHU KOMMOHeHTH. Ilepmmuit 
3 HUX Hoae cTaOiIbHe Apo (PpouTy BOTHIO, a Apyruli - Horo 3MiHHY, BHKJIMKaHy H€BH3HaYeHICTIO CIOCTepexKeHb. 
3alponoHOBaHH MeTOA WO3BOJAE BIATBOPIOBATH LWBUAKOMJIMHHI MPpOCTOpOBO-po3noyiseHl MpowecH Mf iHWIMX KIAaCciB, 
3raPKYIOUN E(PeKTH CIOTBOPEeHb 1 WIYMIB Ip CHOCTepexKeHHI Ta 3a0e3Me4yIOuH MpHMHATHY MPOAYKTUBHICTb. 


Kso4osi copa: Oe3iOTHHH JTaIbHUli allapaT, clocTepexKeHHA 3 OaraTbOX TOOK, HEBH3HaYeHICTb, OOpobKa 
300paxKeHb, (POHT BOTHIO, BIPTYaJIbHi KOMIPKH, Cipa HeYITKa M'AKa MHOXKHHA. 


Introduction class of transient dynamic spatially distributed 
The new millennium has brought the processes of destructive nature, people cannot 
world a significant acceleration of climate build acceptable mathematical models of such 
change, population growth, increased phenomena at the current level of science 
urbanization and industrialization as well as development. Due to the lack of reliable 
many other negative consequences of human models, low observability, and increasing 
activities. The more intense and destructive for scale, speed, and intensity of natural disasters, 
nature is the development of human the tasks of localization, response, and 
civilization, the more nature responds to it by elimination the consequences of disasters 
sufficient changes. Currently, the world is become much more difficult for those involved 
overloaded with natural disasters, from large- because they must be solved with a high 
scale floods and forest fires to sudden degree of responsibility, essential time 
earthquakes, tsunamis, tornadoes, and other constraints in the conditions of uncertainty and 
natural features, which are often lined up in unpredictability. Since response operations are 
chains and accompanied by man-made often based on visual observations, 
disasters because of anthropogenic factors, information support for decision-makers is 
human inattention, negligence etc [1]. crucially actual and therefore is the topic of 
Today, humanity has not adapted to our current interest. 
climate change and has not created adequate 
methods for forecasting and preventing natural Problem statement 
disasters, so it is mostly concerned with the The new era has brought humanity not 
localization and elimination of _ their only new challenges but also new 
consequences. Unfortunately, most of such opportunities related to the development of 
phenomena are random, non-stationary, unmanned vehicles, remote sensing, image 
nonlinear, and unpredictable processes, which, processing, machine learning, and _ other 
moreover, develop very quickly in space and modern technologies. Unmanned aerial 
time and usually are not well observed [2]. vehicles (UAVs) can best assist in monitoring 
However, the consequences of — such the development of destructive processes 
phenomena - destruction of infrastructure, "from above", which allows, on the one hand, 
economic and = environmental damage, to get as close as possible to the observed 
degradation of natural objects, lost health, and events, and on the other hand, keeping them at 
even human lives - are very well observed. a safe distance [3]. The ability to observe 
Thus, given the fact that the considered "above" and "sideways" by UAVs allows a 
natural phenomena can be represented by a much wider coverage of spatially distributed 
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events, reaching a fundamentally different 
scale of situational awareness of decision- 
makers. Therefore, the use of UAVs is 
considered promising in many emergency 
monitoring systems, such as forest fires. 

This paper will be devoted to the 
reconstruction of the forest fire development 
model as one of the most common classes of 
transient destructive processes. However, the 
opportunity of using UAVs in a broader sense 
is provided only by their equipping of certain 
sensors that allow assessing the value of 
certain measured parameters at certain points 
in the observation space. For example, optical 
and infrared cameras are relevant for forest 
fire monitoring, while level, humidity sensors 
are required for flood monitoring. Therefore, 
due to the above-mentioned advantages, UAVs 
equipped with relevant remote sensors can be 
a modern tool for monitoring a wide class of 
transient spatially — distributed _ natural 
phenomena. 

On the other hand, the observation of 
natural phenomena even by UAVs is always 
complicated by obstacles and distortions, so 
the accuracy of remote measurement of 
parameters needed is currently not ideal. For 
example, the effects of wind, smoke and fire, 
air turbulence, and UAV vibrations prevent 
accurate observing the forest fire propagation. 
In addition, there are certain spatial 
constraints, such as the multiplicity of fire 
spots, fire front segmentation, fuel and terrain 
variability, which makes it difficult to obtain a 
reliable image of the situation for planning and 
implementation of fire response operation by 
decision-makers. 

An important advantage of a UAV is that 
it can observe the process moving, i. e., 
changing the point of view of the situation, 
and can approach a certain target point much 
closer than a human. However, due to the high 
combustion temperature, the observation point 
cannot be closer to the forest fire front than it 
is acceptable for UAV safety reasons, so it is 
impossible to overcome the uncertainty of 
observations and capture the process 
completely. Thus, the peculiarities of spatially 
distributed transient processes do not allow to 
fully observe the process from one observation 
point, even in the case of continuous UAV 
movement, therefore, the solution of the 
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process reconstruction problem requires 
simultaneous observation from _ several 
different points [4]. When using multiple 
UAVs, the completeness of the observations is 
significantly added, but a problem of their 
incompatibility and inconsistency occurs. 

It is known that decision-makers must be 
situationally informed to plan response 
operations, i. e., they must understand the 
process development, assess the directions and 
speed of fire spreading. Fortunately, they do 
not require over-detailed information but 
should be informed on the strength, directions, 
and speed of the fire front in time. Thus, when 
reconstructing a process by UAV surveillance, 
the most important limitation is the time 
required: the sooner the dynamics of the 
process are determined, the sooner the 
decision-maker will be able to plan a response 
operation. 

Therefore, the problem addressed in this 
paper relates to the reconstruction of dynamic 
processes using multi-view observation by a 
group of UAVs. We will investigate the 
uncertainty conditions of joint remote sensing 
by multiple UAVs and the relevant issues. 


Related works 

There is always a certain determinant in 
each process that reflects its dynamics. For 
example, in the forest fire process, such a 
determinant is the fire front, which is formed 
under the influence of high-temperature 
combustion reaction and moves, leaving 
already burned areas where there is no fuel left 
for the combustion reaction, and covering new 
areas where fuel is available [5]. The stronger 
the fire, the higher and wider the fire front. 
Thus, the reconstruction of the forest fire 
process is a three-dimensional reproduction of 
the fire front according to the observations of 
fire signs at certain points in space, namely 
fire, flame, smoke, high temperature [6]. The 
presence of the first three signs can be 
recognized using optical cameras, and the 
fourth - using infrared cameras. In any case, it 
is necessary to use modern methods of 
computer vision and remote sensing to assess 
the fire front surface (contour) captured by 
sensors. This task is quite similar to the object 
recognition task. A three-dimensional model 
of an object can be reconstructed by scanning 
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its surface from several viewpoints 
sequentially. A detailed review of the object 
recognition methods is presented in [7]. 
Clearly, objects are usually implied static, but 
forest fire is an entirely dynamic process. 
Thus, well-known object recognition methods 
are generally insufficient [8], however, they 
provide a possibility of sequential recognition 
of the fire front position and state obtaining a 
model of its motion that reflects a process of 
fire propagation. The multi-view object 
reconstruction method has been proposed to 
overcome visibility constraints [9]. This 
approach allows scanning objects in real-time 
but is quite sensitive to sparse and uncertain 
observations. 

A volumetric approach is almost closest 
to the subject of our consideration [10]. There 
are real 3D volumetric methods that describe 
objects by 3D point clouds, and 2.5D methods 
that describe only measured height (or depth) 
for a certain 2D area. Both 3D and 2.5D 
methods are oversensitive to sensor noises and 
cannot distinguish suspicious areas [11]. 
Obviously, a flame is a certain “eigenfeature” 
of the fire front with variable shape and color. 
Effects of smoke, wind, flares, and flickers, 
complicate the flame recognition significantly. 

Thus, various distortions, interferences, 
and obstacles lead to many unrecognized and 
uncertain elements in the scanned image, so 
we need to assess some signs of fire (flame, 
smoke etc.) within certain areas by a degree of 
confidence based on the — uncertain 
observations. The volumetric 3D method 
proposed in [6] uses a soft-rough uncertainty 
model to describe the fire front by a cloud of 
blurred 3D cells. Another approach is 
probabilistic, which represents uncertainty in 
terms of chances [12] but can deal only with 
stochastically stable data. That is why various 
non-probabilistic methods have been 
developed for image recognition tasks. The 
most widely used are fuzzy sets [13], which is 
a convenient tool to represent vague data by a 
membership degree of a certain element. 
Rough set [14] is also a tool to deal with 
imprecise or noisy data based on a relation of 
their indiscernibility, which allows 
approximating it by two crisp sets (lower and 
upper approximations). 

Fuzzy sets mainly represent subjective 


314 


uncertainties highlighting the vagueness of 
information while rough sets mostly represent 
objective uncertainties focusing on incomplete 
information. Both approaches complement 
each other perfectly and are therefore often 
combined. However, both have a common 
disadvantage - they need a priori information 
about the membership function or the 
indiscernibility relation, which cannot be 
obtained in advance during the reconstruction 
of the dynamic process. 

A soft set [15] is a model of both 
uncertainties and vagueness determined 
through a certain parametrization of any kind 
that is free from a priori information 
inadequacies. Grey numbers [16] can also be 
an adequate model in cases when the exact 
value is not completely known. It introduces a 
certain set of values like intervals but 
represents only one of them. 

Usually, the reconstruction of the forest 
fire front is performed under uncertain 
conditions of various nature, which cannot be 
modeled by incomplete, inaccurate, or vague 
information separately. Degrees of confidence 
are spatially distributed and dynamic, while 
the observation space is sufficiently sparse and 
contains many uncertain and suspicious areas. 
Since fuzzy, rough, soft, and grey models are 
closely related and complementary to each 
other [17], it is appropriate to combine them 
into a certain complex model reflexing both 
objective and subjective uncertainty. 

Thus, despite the overall progress in the 
fields of object reconstruction and image 
processing in the conditions of uncertainty, 
many of their issues have a little reflection in 
the literature, remain insufficiently 
investigated, and need further research. 


The aims of the paper 

Considering the above-mentioned 
reasons, the process reconstruction task must 
be solved using three-dimensional 
(volumetric) multi-view observations taking 
into account their incompleteness, uncertainty, 
and inconsistency. Thus, the most topical issue 
for today is the investigation of approaches to 
the 3D reconstruction of dynamic processes 
based on the example of forest fires. The paper 
aims to propose a proper method for 3D 
reconstruction of the forest fire front based on 
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uncertain multi-view observations by a group 
of UAVs considering their distortions, 
interferences, and noises. 


Methodology 

The main idea is that the signs of fire 
(flame, smoke etc.) observed in certain spatial 
areas can be assessed by certain degrees of 
membership in the corresponding class of 
spatial areas. Since the observation space is 
sparse and _ observations are mainly 
incomplete, various uncertain, undefined, or 
suspicious areas can also be approximated by 
certain degrees of confidence. Due to the 
simultaneous effects of several uncertainties of 
different nature, we need to combine fuzzy, 
soft sets, and "gray" numbers in the complex 
uncertainty model. 


A. Reconstructed scene 
The multi-view observation of the forest 
fire by a group of UAVs is depicted in fig. 1. 


Fig. 1. Multi-view fire front monitoring 


Clearly, the fire front can be determined 
by signs of flame. However, the smoke masks 
the flame, and the wind makes them dynamic. 

The reconstructed scene is defined as a 
configuration of poses as shown in fig. 2. The 
pose is a viewpoint, in which spatial position 
is determined by geolocation of UAV and 
orientation of its sensors. The last is usually 
chosen according to the horizontal and vertical 
normal to the image plane. Each pair of poses 
enables a stereoview. 


Based on the known UAV’s geolocation 
in each stereo-pair and relative geometry of 
their poses, the depth of the scene can be 
perceived and, accordingly, the geolocation of 
spatial elements observed from the specified 
poses can be evaluated. A certain spatial 
model can be used to define spatial elements. 
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Fig. 2. The reconstructed scene 


B. The Spatial model 

Discrete spatial models are most often 
used to solve recognition tasks; in the case of 
three-dimensional objects recognition, three- 
dimensional Euclidean space must _ be 
discretized by cubic cells. Thus, we can 
traditionally use isometric cubic cells d,,. 


having the size 6xdx6. Consequently, the 
terrain where the forest fire spreads can be 
represented by three-dimensional space C 


discretized by a grid D={d,,.\ of cells d,,. 


indexed along the axis X , Y, and Z. 

The simplicity of this approach has a 
downside causing the main drawback. Since 
the fire front moves, leaving already burned 
areas where there is no fuel left for the 
combustion reaction and covering new areas 
where fuel is available, the grid D should be 
extended with new rows and_ columns 
dynamically every time as the fire front covers 
new areas. As well, rows and columns, which 
correspond to the burned-out areas, should be 
removed from the grid. In general, this poses 
two problems. First, we need to build an initial 
space of the maximum possible size to provide 
the possibility of developing a_ spatially 
distributed forest fire process to the maximal 
boundaries, which are not initially known. 
There is not only a problem of limiting the 
memory amount here but also the problem of 
significantly reducing performance due to 
excessively large dimensions of space. 
Second, each cell requires geo-localization and 
geo-reference during image processing that 
slows down the fire front reconstruction at a 
high rate of fire propagation, which often 
happens. The higher the accuracy of the 
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sensors, the smaller the cell size, and the 
greater the drop in performance due to their 
excessive number. 

To resolve the contradiction between the 
accuracy of the fire front reconstruction 
process and its speed, we need to build a 
spatial model that contains mainly the set of 
cells, which currently correspond to the spatial 
location of the fire front or close areas. 
Obviously, this can significantly reduce the 
number of cells to be processed. To do this, we 
can organize cells not only in the grid D but 
also in a virtual hierarchical structure J well- 
known as octree (fig. 3) [18]. 


2™4 level of hierarchy 
at 
= \<——- 1s! level of hierarchy 


0 level of hierarchy 


voxel 


Fig. 3. Octree-based spatial structure [18] 


An octree J is based on the principle of 
recursive eight-fold subdivision of spatial 
areas and represents a certain octant of nodes 
at each k -th level of the hierarchy, which can 
be defined by a tuple J, =(1,,G,,<,), where 
G, is a set of nodes, <, is a partial inclusion 
order over G,, and L, is the least element of 


<,. Then, each node g,¢G, represents a 
virtual box as a certain cubic volume. Splitting 


some node g, of level k recursively, we can 


define another octant of sub-nodes (e.g., sub- 
boxes) of smaller size at a lower (k-1) level 
of the octree 7. Such recursion can be applied 
top-down or bottom-up. If 0 is the lowest level 
of J, a smaller virtual box g,¢G, can be 


represented by a certain cell d,,. of the grid 


D. To implement hierarchical structure J, we 
use a revised version of the OctoMap library, 
in which all nodes of the Octree have been 
supplemented by a feature vector (Fig. 4), 
which presents bitwise the generalized state of 
the nested sub-boxes in terms of signs of fire 
and smoke. 

Hence, we get the opportunity to 
consider the fire front as a 3D set F of virtual 


boxes { 81-8} €G, at the certain hierarchy 


level : of the octree 7, each of which can be 
recursively subdivided down-fold to the cell 
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level. If the fire covers a new area, we add a 
new (z+1) level on the top of J considering 
the :-th level as one of its sub-nodes. If some 
areas have been burnt, the correspondent node 
links can be simply eliminated from /. 


Peesurees] 


Neetures] 


Fig. 4. The hierarchical structure of virtual boxes 


C. State of the virtual box 
Let Q={a,..0,} be a set of possible 


states of virtual boxes (table 1). 


Table 1. States of virtual boxes 


Class Function Correspond to 
Q, Empty “Free of fire” areas 
Areas having signs 
Q, Flame Bue 
7 of flame 
Areas having signs 
Q, Smoke & S18 
: of smoke 
Areas, which are 
O, Burnt 
already burnt 
Vegetation areas 
Os Fuel & 
i ready to burn 
Uncertain or 
. suspicious areas 
(ay Uncertain P 
with respect to 
their burning state 
Occluded areas or 
areas, which have 
QO, Unknown 
not been seen by 
sensors 


The “empty” state (@,) corresponds to 
virtual boxes that lie directly between the 
sensor and the surface represented by flame 
and smoke and define “free of fire” areas. The 
signs of flame (@,) means that the virtual box 
corresponds to a “fire surface” as well as the 
signs of smoke (@,) means attribute the virtual 
box to a “smoke cloud’, both cases with a 
considerable degree of confidence. The 
“burnt” state (@,) corresponds to virtual 
boxes, which contain already burnt surfaces 
and, therefore, cannot be involved in the 
combustion processes in the future. The “fuel” 
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state (@,) corresponds to vegetation-covered 


virtual boxes, which at this time do not 
participate in the combustion processes but can 
be ignited due to readiness of the fuel. The 
areas, which do not contain any flammable 
vegetation, belong to the class @, instead of 


O;. 
The “uncertain” state (@,) corresponds 


to virtual boxes, which cannot be reliably 
classified into the classes of state w,- a, based 


on the data captured by sensors. Such virtual 
boxes are mainly consistent with suspicious 
areas possibly involved in the combustion 
process but there is not enough confidence due 
to the uncertain observations. The “unknown” 
state (@,) corresponds to both virtual boxes 


occluded by boxes of other classes, which 
prevent the perception of the fire front, and 
virtual boxes, which have not been seen by 
sensors. 

Initially, the reconstruction scene is 
represented by a 3D set of virtual boxes 
marked by an “unknown” state. Depending on 
the result of image processing, each virtual 
box must be matched to a certain class w,<«Q. 


Obviously, the virtual boxes of a @,-class 


correspond to clearly observable inner 
boundaries (“burning kernel”) of the fire 
propagation process, the virtual boxes of a a, - 
class correspond to a “smoke cloud”, i. e. the 
areas around the fire front shrouded in smoke, 
which masks “surface” of the fire front and 
prevents its confident recognition, and the 
virtual boxes of @,-class correspond to areas, 


which, for various reasons, are poorly 
observed or about which there are doubts due 
to inaccuracy, inconsistency, or 


incompatibility of observations from different 
viewpoints. The virtual boxes of the classes a, 


and @, represent clearly visible outer 
boundaries of the fire propagation process. 
Blurred, uncertain, or underdetermined areas, 
which dynamically change because of wind, 
smoke, and other factors, lie between the 
clearly observed inner and outer boundaries of 
the fire propagation process. Of course, during 
the reconstruction, all virtual boxes, which 
state cannot be clearly and reliably determined 


in the range @,-@,, are identified as “free of 
fire” (@,), as well as all virtual boxes, which 
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cannot be observed by sensors for various 
reasons, remain in a state “unknown” (@, ). 


D. Image processing 

To classify virtual boxes according to 
table.1, we can use a multi-channel image 
recognition system (fig. 5). 


IR sensor OE sensor 


Image 


averaging 


Grey color 
evaluation 


Image 


mapping 


IR 
Channel 


a es 
|! Smoke i Burnt +; ae i 
1 | Channel H : Channel Channel 
Flame i 1 
:Channel 
i 


Fig. 5. Image processing 


The infrared channel is reliable in 
nighttime and under heavy smoke because of 
the infrared transparency of the latter, but in 
the daytime, it has too small contrast, 
therefore, images can be affected due to 
reflected sunlight, saturation etc. Heat 
radiation can be determined in captured 
images by the color of pixels, which changes 
from black to white depending on the heating 
level. Although combustion is always 
represented by pixels of close-to-white colors 
while free of fire areas are always black, due 
to imprecise, uncertain, or ambiguous 
measuring in the daytime, unfortunately, most 
of the pixels are in different shades of gray. 
Thus, in the infrared channel, image 
processing is performed in three stages: image 
averaging, gray color evaluation, and image 
mapping [5]. At the output of the last stage, 
each defined virtual box g, is associated with 


a certain “degree of grayness” w, calculated 
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based on the average brightness B, of the 


corresponding pixels in the captured image 
and mapped to the range [0, 1]. 

Based on the images captured by optical 
cameras, two main optical channels provide 
flame [19] and smoke [20] recognition 
separately. Both channels work identically 
based on the HS/ color model, but they have 
different settings. In the flame recognition 
channel, the conventional fires correspond to 
colors from dark red to light yellow color with 
high saturation and profiles 
Pp, =[0° - 60°, 0.4 —1.0,0.5—1.0] the 
daytime and p, =[0°—60°,0.2 —0.85,0.5 —1.0] 
in the nighttime or in bad weather conditions. 
The well-developed fires correspond to colors 
from light blue to white with high saturation 
and profiles, respectively, 
P; =[220° — 260°, 0.4 —1.0,0.5 —-1.0] and 
P, =[220° — 260°, 0.2 — 0.85, 0.5 —1.0] 
respectively [21]. In the smoke recognition 
channel, the intensity is high, saturation is low, 
and colors are mainly gray and silver, so the 
profiles are defined, respectively, as 
Ps =[355° —5°,0—0.05, 0.5 — 0.85] and 
Ps =[230° — 240°, 0 — 0.05, 0.5 — 0.85]. For this 


reason, in the flame recognition channel, the 
center colors are bright orange ( z, =[30°,*,*]) 


and blue (, =[240°,*,*]) while in the smoke 


recognition channel the center colors are silver 
(, =[235°,*,*]) and white (2, =[0°,*,*]) 
[22]. 

The other two optical channels are 
auxiliary and developed to recognize burnt and 
fuel-ready areas. They work in a similar way 
to the above-mentioned main channels, but 
their profiles are different. The first of them is 
characterized by low values of intensity (in the 
range [0-0.25]) for any values of hue and 
saturation. 

The second one is determined mainly by 
light green color tones; however, the channel’s 
settings may vary significantly in different 
areas of different climatic zones. In Southern 
Ukraine, the profile 

Pp; =[100° -120°,0.3—0.75,0.1—0.8] is 


the most appropriate. 


in 


318 


In the optical channels, the image 
processing is performed in the following 
sequential stages: 

- image transformation: to convert each 
pixel of the image from RGB to HSI color 
model and normalize all components to the 
ranges 0SH<360°, O<S<1, and O</<1. After 
that, the intensity and saturation components 
are separated from the color components to 
speed up the next stages. 

- image filtering: the pixels, which HSI 
values do not match the proper ranges for the 
filters (p, — ?,), should be eliminated and their 


colors should be replaced by the rose color to 
speed up the next stages. 

- image segmentation: based on the 
color feature and the raster scanning 
algorithm, the image is divided into certain 
rectangular areas (segments) [23]. 

- image averaging: the H, S, and I 
values of pixels are averaged to the mean 
values H,, S,, and J, within each /-th 
segment. 

- image generalization: for each 1-th 
segment, it calculates the degree of similarity 
to the closest center color in the range [0, 1] 
based on the mean values H,, S,, /,. 


1? 


- image mapping: the image is oriented, 
transformed, and then all its segments are 
mapped to the uniquely corresponding virtual 
boxes. 

After the image segmentation stage, due 
to the non-linearity and incompleteness of the 
observations, four-layer fully connected 
backpropagation neuron networks, which 
consists of the input, two hidden, and the 
output layers, are used in image processing in 
four optical channels to improve the accuracy 
and reliability. The network architecture is 
15x18x6x1. The input data is the H, S, and J 
value of pixels of three last consecutive image 
frames (i-1, i, i+1). Besides that, the features 
of bad weather W and daytime D are also used 
as inputs. The transfer functions are defined as 
log-sigmoid at the input and first hidden 
layers, and as the Gauss-based radial basis 
function at the second hidden layer. The 
number of the hidden layers and hidden nodes 
have been experimentally evaluated. The 
output is the degree of similarity of the input 
to the corresponding samples. 


ISSN 2710-1673 Artificial Intelligence 


2022 Nel 


At the final stage, each j-th virtual box 
g, iS connected to a certain confidence vector 


oF = see pores | where yw, is the degree 
of greyness, |, ¢;, ¢;, and % are degrees of 


similarity in respectively flame, smoke, burnt 
and fuel-ready channels. All elements of £, 


are normalized in the range [0, 1]. 
Considering the structure J as a simple 


set of virtual boxes, / ={g ik we define a 
iS je 


ran 


3D observation vector = as a correspondent 


m 


array of confidence vectors, E= iE, 


j=0" 


E. Classification of virtual boxes 
Let B={f,,,,f,} be a set of threshold 


values and A={/,,4,,4,,4,} be a set of factors 


defined such that 4,+ 4, =1 and 4,+4,=1. 


The classification of virtual boxes 
concerning the set of states Q and the 3D 
observation vector = is performed in four 
following stages (fig. 6). 

At the first stage (scene analysis) we 
divide the 3D observation vector = into 
foreground and background areas using the 


threshold values £,¢B. Thus, the background 
area contains all virtual boxes g, that ¢,<£,, 
the foreground area contains all virtual boxes 
g, that ¢,2f,, and the suspicious area 
contains all virtual boxes g, such that ¢, < £,. 


At the second stage (flame confidence 
clarification), based on the set of factors A as 
well as «day/night» and weather features, we 


clarify the flame confidence 7, for each 
virtual box g, related to foreground or 


suspicious areas. In the daytime and under 
good weather conditions, the foreground cells’ 
flame confidence is 7,;=¢, while suspicious 


cells’ flame confidence is 7, =2,-u,+4,°é,. In 


the nighttime and under bad_ weather 
conditions, the foreground cells’ flame 
confidence is 9, =4,°Hj tA §; while 


suspicious cells’ flame confidence is 7, = ;. 
In most cases, the factor’s values 4,=-¢, and 


4,=¢, is convenient. As a result, we 
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substitute the couple w, and ¢, by the flame 


confidence 7, in all confidence vectors 


Image 
processing 


Image 
processing 
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Confidence 
Clarification 


3D-CNN 
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Fuzzy Soft Fuzzification 
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of the Fire 
Front 
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Fire Front 
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Linear 
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Non-blind 
deconvolution 


Fig. 6. Classification of virtual boxes 


At the next stage, we classify virtual 


boxes t={g,}"_ concerning the 3D 
ave 

observation vector = using a 3D 

convolutional neural network (3D-CNN), 


which is a special kind of CNNs based on 3D 
volumes in the kernels instead of 2D maps. 
We use VoxNet [24] 3D-CNN architecture 
(fig. 7). Its convolutional layers are based on 
128 features, 5x5x5 kernel, 2x22 stride, and 
2x2x2  max-pooling operation (Cl), 32 
features, 4x4x4 kernel, 2x2x2 stride, and 
2x2x2 max-pooling operation (C2), 8 features, 
3x3x3 kernel, 2x22 stride, and 2x22 max- 
pooling operation (C3). Fully connected layers 
FC1, FC2, FC3 are based on 1200, 400, and 32 
parameters respectively. The output is defined 


as a confidence vector C;, =(a penal pe where 


Qi 


box g; is in the state o,, a, e[0,1]. 


is a degree of confidence that the virtual 
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Convolutional Layer 3 
1@8x3x2 + P(2) 


6x6x6 

Full Connected Layer 1 

Input volume ay) 

128x128x128 

Full Connected Layer 2 

(400) 

Convolutional Layer 1 Convolutional Layer 2 
3@128x5x2 + P(2) 1@32x4x2 + P(2) Full Connected Layer 3 
16x 16x16 12x12x12 (32) 


eek . 

Let ®={9,}, be a family of fuzzy sets 

g,, each of which contains all virtual boxes 
g, <1, the state of which is @,. 


At the final stage, we do fuzzification to 
decompose the output of 3D-CNN into fuzzy 
sets of a family ®. Thus, using the triangular- 
shaped membership functions, the degrees of 
membership to which the certain virtual box 
g, belongs to each given fuzzy set g,<¢® 


should be determined. 


F. Reconstruction process 

Let U be a set of UAVs jointly 
observing the terrain, where a forest fire 
dynamic process is developing. Suppose each 
UAV U,«€U is located at a distinctive 


viewpoint. Let T be a linear ordered timescale 
such that ¢,,,, >¢,, . 

Let I be a spatial model of the 
considered terrain represented by the 
hierarchical structure of virtual boxes denoted 
by g,,:, where / is the level of the virtual box 


&,,, mm the hierarchy J, qg is a connection to 


the comprising virtual box of level /+1, and i 
is a number of the virtual box within the 
comprising virtual box (from | to 8). 

Let Vii be a degree of membership to 
which the /gi-th virtual box belongs to the 
fuzzy set @, of virtual boxes being in the state 
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Output 
(7) 


Fig. 7. 3D-CNN Architecture 


o, from the viewpoint of U,<«U at the 


moment teT. 
Thus, a fuzzy state g,,,, of the virtual 


box g,,; visible from UAV U, at the moment 
t 1s a vector 

Bur = (Fries )-(Figirs)) > CD) 
where g,,, and /,,;,, are defined as above, 


Q,,..0, €Q. are the states defined in table 1. 


Since the purpose of multi-view 
observation is to capture images of the target 
same areas from different positions 
simultaneously, several UAVs are watching the 
process development differently from different 
viewpoints. Certain areas of the terrain are 
better visible to one UAV, but others see it 
worst, so the first one has more confidence 
than others. Moreover, their information can be 
inconsistent; the accuracy and completeness of 
observation can also vary. Given the fact that 
UAVs most often move during the observation, 
the information is usually dynamic, so the 
confidence and accuracy of observation may 
increase at some points while decreasing at 
other points. 

Clearly, we need a reasonable way to 
combine uncertain observations of the same 
virtual box from different viewpoints and 
clarify them during continuous observation. A 
hesitant multi-fuzzy set [25] can be used to 
combine observations from different observers 
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U,,..U,,. Thus, a multi-view fuzzy state G8 of 
the virtual box g,,; by UAVs U,,..U,, at the 


moment ¢ can be represented as 
~ (Bs Vaiss Veni) 9 


lqit i ~ = 
(o, aCe ote Vigintt )) 
There is a 7-elements array of classes 


(2) 


o,, 


each of which is defined by the correspondent 
fuzzy set of virtual boxes g, and represented 


by an array of n degrees of membership to this 
set concerning n different observers. All these 
membership degrees lie on the interval 
between their minimum and maximum values. 
Such intervals can be consequently narrowed 
during continuous observations. Thus, (2) can 
be transformed to: 


min( lqgillt? + iginit 1 
D> 3 


MAX | Yigitieo-+-Yiginit ) 


° 


B igit = : a = (3) 
mun (Pies eV igin Tt ): 
Q,, : ; 
max (Fagin oY igin7t ) 
without loss of accuracy. 
Suppose Baie = min (7 Igitkt ) and 
Brains = max (Fu )> &=[1L7]. The  above- 


mentioned intervals can be further defined as 
Diag = [Bais Bons . (4) 
Such definition corresponds exactly to 


grey numbers [16]. Given this fact, we can 
define a grey fuzzy state of the virtual box g,,, 


at the moment ¢t based on (3) and (4) as: 
= + 7 
8 igi = ((o. »Braitt Me (5) 


Let > be a partial order (preference) 
relation induced on the set of possible states Q 
of virtual boxes, such that 
0, > 0, > O,>O,>@;>@,>oa,. This relation 
allows determining the class i of the virtual 
box in the case of approximate equality of 
some membership degrees by the selection of 
the most trusted of them: 


fi = (@., Bir) if @, >all other o,. 


G. Fire front representation 

Let the virtual structure 7 be a universe. 
Suppose Q is a parameter set. Let Y be a 
mapping of © into a set of all possible subsets 
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of virtual boxes of J. A pair (Y,Q) can be 


considered as a soft set Y of virtual boxes 
over J [15]. In this case, the soft set Y =(Y,Q) 
is a family of certain subsets of the 
hierarchical set of boxes J parameterized by 
the elements of the set Q, i. e., by the specific 
states of the virtual boxes. Hence, Y(q,,t) is 
considered as a w, -approximate element of the 
soft set of the virtual boxes being in the state 
@, €Q at the moment teT. Therefore, Y is a 
dynamic mapping and the correspondent soft 
set is also dynamic, so it can represent the 
dynamics of the observed process. We regard 
the soft set Y consisting of 7 elements as 
below: 


Ye {(@.¥ (@,.t)) :@, €2°,Y(@,,t)€ ay 
Suppose the grey number /; defined as 
[ Pes al represents the degree 7, of the 


virtual box membership in the @, -approximate 
element of the soft set Y, i. e. the set of the 
virtual boxes being in the state @, €Q at the 
moment of the process reconstruction te7. In 
this case, a mapping of Q into the set of 
virtual boxes of J must take into account their 
membership degree, so the elements of the soft 


set can be defined as Y(@,t,/,,)— ([0.1].[0.1]) 
and the pair (Y.9) defines a soft grey fuzzy 


set Y. 
Let ste) 


Tiz>T €[O.1], 7, <7,. Using these numbers as 


be a pair such that 


thresholds we can divide each element of the 


soft grey fuzzy set Y into certain components. 
With respect to the w, -element of the soft grey 


fuzzy set Y, the first component is a lower 
approximation Y, (t)=Y (@,,t,[7,, 27, ]) that 


=k 
contains the virtual boxes definitely belonging 
to this element, the second one is an upper 


approximation y, (t)=Y(o,.t[7,,27,]) that 
contains virtual boxes most likely belonging to 
this such that y, (t)<¥,(t), the 
difference 4 (t)=Y, (¢)- 4 (t) between them is 
a vague boundary that contains doubtful virtual 
boxes, and Y, (t)=Y(@.t[%,.<t,]) is a 


negative region that contains virtual boxes, 


element, 
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which most likely do not belong to the o,- 


element. 

The fire front F is defined using the 
following assumptions. Due to the dynamics 
of the forest fire and the effects of wind, 
terrain, and a wide range of distortions, 
interferences, and noises the observed states of 
the virtual boxes are dynamic and principally 
uncertain. Therefore, the fire — front 
reconstruction can be performed by the 
uncertain distribution of the virtual boxes over 
the following components. 

The = convincing — component F* 
represents the stable kernel of the fire front 
defined by the lower approximation of the , - 


element Y, (t) of Y, F*(t)= Y, (t). 

The negative component F contains 
virtual boxes, which should be confidently 
excluded from the fire front during the 
reconstruction, and is defined as the union of 
the lower approximations of the @,-, @,-, @,- 
O6- of ye 
F (t)=¥,(t)UY, (UY, (t)UY,(t)UY (t). In 
order to reduce the computations, @,- and a, - 


and o, -elements 


elements may not be approximated, then 
F (t)=¥,(t)UY, (UY, (t)UY,(t)UY, (st). If 
necessary, the same simplifications may apply 
to the w,- and @,-elements, moreover, the @, - 
and @,-elements may not even be grayed. In 
this case, 
F (t)=¥,(t) UY, (1) UY, (1)UY, (1) UY, (0). 
The uncertain component F" represents 
the variable component of the fire front 


defined by the union of the vague boundaries 
of the @,- and @,-elements and the lower 


approximation of the «@,-element of Y, 
F*(t)=¥,(t)U¥,(1)UY, (2). 
The suspicious component F' of the 


fire front is defined by the vague boundaries of 
the @,-, @;-, @,-, @,-elements, and, if 


applicable, Q,- element of 
F =¥(1)UY,()UY,(NUY,()UY,(). 

Thus, the soft grey fuzzy model of the 
fire front F can be defined as 
F=F'*UF' VF UF. 


A 


Y, 
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Let At=t,,,-1,, ¢,,¢,,¢T be a step of 
observation. Assume that the noise model is 
Poisson. Suppose the initial kernel of the fire 
front is determined by a __ statistically 
continuous part of the convincing component 


F*(t). Since the fire front is vague and not 
stable, we use a linear’ time-invariant 
convolution [26] to find its stable kernel F° 
with a certain latent Z and noise N. Thus, 
F*(t)=F°@L+N. The gradient method 


increases the kernel after each next step, be 
used. Then, a non-blind deconvolution method 
[27] can be performed to improve the kernel 
determined in the previous step. 

The pair (7z,,.7,,) sets certain thresholds 
on the confidence for each ,-element of the 
soft grey fuzzy set Y and can vary due to the 
dynamics of the process. Their values can be 
determined during the continuous observation 
within each subsequent step by the integral of 


the minimum and maximum values, i. e., 
1, +At t,+At 
Ty = | min (Fyn) Ty = max (Fin )- 
t 1; 
Implementation 
The proposed method has _ been 


experimentally implemented using Visual C++, 
OctoMap framework, ConvNet, and FANN 
libraries. The software prototype has been 
tested on a PC network (Pentium i17-10700 2.9 
GHz, 32 Gb RAM, 512 Gb SSD). 

Computer simulation results show that 
the proposed method allows reconstructing the 
fire front during multi-view observation by 
several UAVs representing its stable core 
(flame) in red colors, the variable component 
in shades of orange, the smoke in pink colors, 
the uncertain components in yellow, and the 
negative component in shades of blue. Fig. 8 
depicts a visualization of the fire front. 

The accuracy and the rate of the 
reconstruction has been evaluated during the 
experiment. Fig. 9 shows the accuracy of the 
reconstruction depending on the number of 
observers (UAVs) and the scale of the terrain 
model represented by the average size of the 
least virtual boxes. Fig. 10 shows the average 
time of the reconstruction of the fire front 
depending on the same parameters. 
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Fig. 8. Visualization of the forest fire front 


Based on the experimental results, we 
concluded that the proposed method of the 
volumetric reconstruction of the fire front 
provides enough credibility to the decision- 
maker and sufficient performance to meet the 
requirements of the dynamic process 
reconstruction in real-time. 


Accuracy vs average size of the least virtual boxes 


[S eee} ObSEVErs 


Fig. 9. Evaluation of Accuracy 


Reconstruction time vs everage size of the least virtual boxes 


Fig. 10. Evaluation of Reconstruction time 


Conclusion 

The paper presents a novel method of the 
volumetric reconstruction of the forest fire 
front. based on _ uncertain multi-view 
observations by a group of UAVs. The 
proposed method provides high performance 
due to a high-speed spatial model based on a 
virtual box hierarchy, which allowed to 
exclude iterative geo-localization and 
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geo-rectification processes from the main 
image processing stage. The fire front 
representation is based on the fuzzy, soft sets, 
and grey numbers, which allowed to smooth 
the effects of distortions, interferences, and 
noises during the observation. The proposed 
approach ensures the possibility to reconstruct 
other classes of transient spatially distributed 
destructive processes if UAVs are equipped 
with suitable sensors and if the adequate 
classification of the spatial elements’ states is 
defined. Further studies will be devoted to the 
blurring of spatial elements to improve the 
compatibility of multi-view observations and 
the optimization of the scene concerning the 
optimal location of observers to provide a 
more accurate reconstruction of the process. 
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